Skip to content

fix: apply native reasoning effort to harnesses - #54

Open
vincentkoc wants to merge 2 commits into
mainfrom
fix/native-reasoning-effort-parity
Open

fix: apply native reasoning effort to harnesses#54
vincentkoc wants to merge 2 commits into
mainfrom
fix/native-reasoning-effort-parity

Conversation

@vincentkoc

@vincentkoc vincentkoc commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What does this PR do?

Propagates the native eval plan's reasoning effort into OpenClaw, Hermes,
Codex, and Claude Code client controls.

Why?

Fixes #53. Without this, a run can be labeled high while OpenClaw executes
with thinking disabled, invalidating cross-harness quality and efficiency
comparisons.

Changes

  • carry reasoning_effort in RunSpec and the run manifest
  • map effort to each native harness's supported control
  • translate ShellBench's canonical xhigh to Claude Code's max
  • preserve existing harness defaults when effort is unspecified
  • test low, medium, high, and xhigh across all four harnesses

Tests

  • 114 focused native runner, proxy, and fleet tests pass
  • focused Ruff check passes
  • fresh Codex autoreview reports no actionable findings
  • all GitHub CI and CodeQL checks pass

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 Urgent regression or broken agent/channel workflow affecting real users now. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. labels Jul 29, 2026
@clawsweeper

clawsweeper Bot commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed August 5, 2026, 4:51 PM ET / 20:51 UTC.

ClawSweeper review

What this changes

The branch carries planned reasoning effort into native benchmark commands and manifests, mapping it to OpenClaw, Hermes, Codex, and Claude Code with regression coverage.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep open for maintainer review: the PR is authored by a repository member and provides a focused, source-supported repair for the linked benchmark-parity bug.

Priority: P1
Reviewed head: 887c2597c1e6ef7ad369cd9a72c800b8fb52ffeb
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) The patch is focused and source-supported with comprehensive mapping assertions; native-client compatibility remains the material merge check.
Proof confidence 🌊 off-meta tidepool Not applicable: This repository-member PR is not subject to the external-contributor real-behavior-proof gate; focused tests and CI are supplemental evidence.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Not applicable Not applicable: This repository-member PR is not subject to the external-contributor real-behavior-proof gate; focused tests and CI are supplemental evidence.
Evidence reviewed 4 items Current-main mismatch: Current main hard-codes OpenClaw thinking off, matching the linked report's concrete parity failure.
Proposed propagation: The branch derives the OpenClaw setting from planned effort, adds Hermes and Codex controls, and maps canonical xhigh to Claude Code max.
Focused regression coverage: The branch tests all four harnesses at low, medium, high, and xhigh, plus the no-effort defaults.
Findings None None.
Security None None.

How this fits together

ShellBench converts evaluation plans into native-client commands and records each run’s metadata. These commands determine how each harness executes benchmark tasks through the proxy, so consistent reasoning settings are needed for comparable results.

flowchart LR
A[Evaluation plan] --> B[Run specification]
B --> C[Harness command builder]
C --> D[Native client control]
D --> E[Benchmark task run]
B --> F[Run manifest]
E --> G[Comparable results]
Loading

Decision needed

Question Recommendation
Should the pinned OpenClaw, Hermes, Codex, and Claude Code versions be accepted as the supported reasoning-effort contract for this benchmark suite? Validate and merge: Run representative commands against each pinned client version and merge when every control is accepted and reflected in the run output.

Why: The patch’s command construction is coherent, but the final compatibility contract spans independently versioned clients and this member-authored PR requires explicit maintainer handling.

Before merge

  • Resolve merge risk (P1) - The patch emits controls consumed by separately versioned native clients; an unsupported flag or configuration key could stop a harness instead of yielding a comparable benchmark run.
  • Complete next step (P2) - Explicit maintainer validation of the pinned native-client contract is required before merge.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Patch surface production +29/-3, tests +92; 4 files affected The implementation is narrowly concentrated in native command construction, run metadata, and its regression tests.

Merge-risk options

Maintainer options:

  1. Verify pinned-client compatibility (recommended)
    Capture redacted output from one representative run per native client before merge to confirm every emitted control is accepted.
  2. Accept the version-contract risk
    Merge based on focused command-construction tests while explicitly accepting compatibility responsibility for the pinned toolchain.

Technical review

Best possible solution:

Validate the four mappings against the pinned native-client versions, then land the parity fix so planned, applied, and recorded reasoning effort agree.

Do we have a high-confidence way to reproduce the issue?

Yes—current main hard-codes OpenClaw thinking off while the plan can specify high effort, and the branch’s focused tests exercise the intended command mappings.

Is this the best way to solve the issue?

Yes—the branch uses the existing run-spec and environment flow rather than adding a parallel configuration path; pinned-client validation is the remaining safety check.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

  • P1: A reasoning-mode mismatch can invalidate active cross-harness quality, latency, and efficiency comparisons.
  • merge-risk: 🚨 compatibility: The change adds client-specific flags and configuration keys across four separately versioned native tools.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This repository-member PR is not subject to the external-contributor real-behavior-proof gate; focused tests and CI are supplemental evidence.

Evidence

What I checked:

Likely related people:

  • Vincent Koc: Authored the current-main native-harness implementation and the two commits that add reasoning-effort propagation and Claude Code normalization. (role: recent native-evaluation contributor; confidence: high; commits: 884dd1bb5511, 98301b240507, 887c2597c1e6; files: scripts/native_eval/harnesses.py, scripts/native_eval/models.py, scripts/native_eval/run_job.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Provide redacted representative output for the pinned OpenClaw, Hermes, Codex, and Claude Code clients.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (36 earlier review cycles; latest 8 shown)
  • reviewed 2026-08-04T18:03:27.270Z sha 887c259 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-04T21:07:16.018Z sha 887c259 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-04T23:03:06.433Z sha 887c259 :: needs changes before merge. :: [P1] Preserve reasoning effort during fleet reconstruction
  • reviewed 2026-08-05T00:18:26.093Z sha 887c259 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-05T08:36:33.545Z sha 887c259 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-05T12:08:13.503Z sha 887c259 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-05T12:31:32.134Z sha 887c259 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-05T19:41:05.675Z sha 887c259 :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot mentioned this pull request Jul 29, 2026
Open
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 29, 2026
@vincentkoc
vincentkoc marked this pull request as ready for review July 29, 2026 16:23
@vincentkoc
vincentkoc requested a review from a team as a code owner July 29, 2026 16:23
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 29, 2026
@clawsweeper clawsweeper Bot added status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P1 Urgent regression or broken agent/channel workflow affecting real users now. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Native harnesses do not apply planned reasoning effort consistently

1 participant

@vincentkoc